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I.  RESEARCH  ACCOMPLISHMENTS 

Research  for  ONR  Grant  "Statistical  Inference  From  Sampled  Data"  has  been  concerned  in  the  past 
8  years  with  studies  in  the  general  area  of  statistical  signal  processing.  Contributions  were  made  in  a  wide 
range  of  topics  motivated  by  practical  problems  in  communication  systems  and  digital  signal  processing. 
The  following  is  a  list  of  the  main  topics  on  which  the  research  was  focused: 

A.  hiference  for  Continuous-Ume  Processes  from  Sampled  Data. 

B .  Wavelets,  Signal  Approximation,  and  Function  Estimation. 

C.  Neural  Networks  and  Function  Estimation. 

D.  Probability  Density  and  Regression  Estimation  from  Noisy  Dependent  Data. 

E.  Nonlinear  Time  Series. 

F.  Local  Polynomial  Fitting. 

G.  Spread  Spectrum  Communication  Systems. 

H.  Other  Contributions. 

The  research  under  this  grant  resulted  in  the  publication  of  32  refereed  papers  in  mathematical, 
statistical,  and  engineering  journals.  In  addition,  several  papers  were  presented  and  subsequently 
published  in  proceedings  of  conferences.  Copies  of  these  works  were  routinely  sent  to  the  Office  of  Naval 
Research,  Surveillance,  Communications,  and  Electronic  Combat  Division.  The  following  is  a  list  of  the 
journal  publications  under  this  grant: 

1.  L.  Gyorfi  and  E.  Masry,  "The  Lx  and  L2  Strong  Consistency  of  Recursive  Kernel  Density 
Estimation  From  Dependent  Samples,"  IEEE  Trans.  Information  Theory,  Vol.  IT-36  (May  1990),  pp. 
531-539. 

2.  E.  Masry,  "Almost  Sure  Convergence  Analysis  of  Autoregressive  Spectral  Estimation  in  Additive 
Noise,"  IEEE  Trans.  Information  Theory,  Vol.  IT-37  (January  1991),  pp.  36-42. 

3.  E.  Masry,  "Multivariate  Probability  Density  Deconvolution  for  Stationary  Random  Processes,"  IEEE 
Trans.  Information  Theory,  Vol.  IT-37  (July  1991),  pp.  1105-1115. 

4.  E.  Masry  and  J.  A.  Rice,  "Gaussian  Deconvolution  via  Differentiation,"  Canadian  J.  Statistics,  Vol. 
20,  No.  1  (1992),  pp.  9-21. 


5.  E.  Masry,  "Flicker  Noise  and  the  Estimation  of  the  Allan  Variance,"  IEEE  Trans.  Information 
Theory,  Vol.  IT-37  (July  1991),  pp.  1173-1177. 

6.  S.  Cambanis  and  E.  Masry,  "Trapezoidal  Stratified  Monte  Carlo  Integration,"  SIAM  J.  Numerical 
Analysis,  Vol.  29  (February  1992),  pp.  284-301. 
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7.  K.  S.  Lii  and  E.  Masry,  "Model  Fitting  for  Continuous-Time  Stationary  Processes  from  Discrete- 
Time  Data,"  J.  Multivariate  Analysis,  Vol.  41  (April  1992),  pp.  56-79. 

8.  E.  Masry,  "Asymptotic  Normality  for  Deconvolution  Estimators  of  Multivariate  Densities  of 
Stationary  Processes,"  J.  Multivariate  Analysis,  Vol.  44  (January  1993),  pp.  47-68. 

9.  J.  Fan  and  E.  Masry,  "Multivariate  Regression  Estimation  With  Errors-in- Variables:  Asymptotic 
Normality  for  Mixing  Processes,"  J.  Multivariate  Analysis,  Vol.  43  (November  1992),  pp.  237-271. 

10.  C.  Lo,  E.  Masry,  and  L.  B.  Milstein,  "Design  and  Analysis  of  a  Fast  Frequency-Hopped  DBPSK 
Communication  Systems,  Part  I:  System  Description  and  Hop  Timing  Tracking  Loop  Analysis," 
IEEE  Trans.  Communications,  Vol.  COM-41  (October  1993),  pp.  1552-1564. 

11.  C.  Lo,  E.  Masry,  and  L.  B.  Milstein,  "Design  and  Analysis  of  a  Fast  Frequency-Hopped  DBPSK 
Communication  Systems,  Part  U:  Error  Performance  in  AWGN  Plus  Partial-Band  Noise  Jamming," 
IEEE  Trans.  Communications,  Vol.  COM-41  (November  1993),  pp.  1723-1735. 

12.  E.  Masry,  "The  Wavelet  Transform  of  Stochastic  Processes  with  Stationary  Increments  and  its 
Application  to  Fractional  Brownian  Motion,"  IEEE  Trans.  Information  Theory,  Vol.  IT-39  (January 
1993),  pp.  260-264. 

13.  E.  Masry,  "Strong  Consistency  and  Rates  for  Deconvolution  of  Multivariate  Densities  of  Stationary 
Processes,"  Stochastic  Processes  and  Applies.,  Vol.  47  (August  1993),  pp.  53-74. 

14.  E.  Masry,  "Multivariate  Regression  Estimation  with  Errors-in- Variables  for  Stationary  Processes," 
Nonparametric  Statistics,  Vol.  3  (1993),  pp.  13-36. 

15.  S.  Cambanis  and  E.  Masry,  "Wavelet  Approximation  of  Deterministic  and  Random  Signals: 
Convergence  Properties  and  Rates,"  IEEE  Trans.  Information  Theory,  Vol.  IT-40  (July  1994),  pp. 
1013-1029. 

16.  K.S.  Lii  and  E.  Masry,  "Spectral  Estimation  of  Continuous  Time  Stationary  Processes  from  Random 
Sampling,"  Stochastic  Processes  and  Applies.,  Vol.  52  (August  1994),  pp.  39-64. 

17.  E.  Masry,  "On  the  Truncation  Error  of  the  Sampling  Expansion  for  Stationary  Bandlimited 
Processes,"  IEEE  Trans.  Signal  Processing,  Vol.  42  (October  1994),  pp.  2851-2853. 

18.  E.  Masry,  "Probability  Density  Estimation  from  Dependent  Observations  using  Wavelets 
orthonormal  bases,"  Statistics  and  Probability  Letters,  Vol.  21  (1994),  pp.  181-194. 

19.  E.  Masry  and  F.  Bullo,  "Convergence  Analysis  of  the  Sign  Algorithm  for  Adaptive  Filtering,"  IEEE 
Trans. /M/ormaft'on  r/teo/y,  Vol.  41  (March  1995),  pp.  489-495. 

20.  E.  Masry  and  L.  Milstein,  "Enhanced  Signal  Interception  in  the  Presence  of  Interference,"  IEEE 
Trans.  Communications,  Vol.  COM-43  (March  1995),  pp.  1089-1096. 
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21.  K.S.  Lii  and  E.  Masry,  "On  the  Selection  of  Random  Sampling  Schemes  for  the  Spectral  Estimation 
of  Continuous  Time  Processes,"  J.  Time  Series  Analysis,  Vol.  16  (May  1995),  pp.  291-311. 

22.  E.  Masry  and  D.  Tjostheim,  "Nonparametric  Estimation  and  Identification  of  Nonlinear  ARCH 
Time  Series,"  Econometric  Theory,  Vol.  11  (June  1995),  pp.  258-289. 

23.  E.  Masry,  "Convergence  Properties  for  Wavelet  Series  Expansions  of  Fractional  Brownian  Motion," 
Applied  and  Computational  Harmonic  Analysis,  Vol.  3  (July  1996),  pp.  239-254. 

24.  D.  Modha  and  E,  Masry,  "Rate  of  Convergence  in  Density  Estimation  Using  Neural  Networks," 
Neural  Computation,  Vol.  8  (July  1996),  pp.  1102-1122. 

25.  E.  Masry,  "Multivariate  Local  Polynomial  Regression  for  Time  Series:  Uniform  Strong  Consistency 
and  Rates,"  /.  Time  Series  Analysis,  Vol.  17  (November  1996),  pp.  571-599. 

26.  D.  Modha  and  E.  Masry,  "Minimum  Complexity  Regression  Estimation  With  Weakly  Dependent 
Observations,"  IEEE  Trans.  Information  Theory,  Vol.  42  (November  1996),  pp.  2133-2145. 

27.  E.  Masry  "Multivariate  Regression  Estimation:  Local  Polynomial  Fitting  for  Time  Series," 
Stochastic  Processes  and  Their  Applications,  Vol.  65  (December  1996),  pp.  81-101. 

28.  E.  Masry,  "Polynomial  Interpolation  and  Prediction  of  Continuous-Time  Processes  From  Random 
Samples,"  IEEE  Trans.  Information  Theory,  Vol.  43  (March  1997),  pp.  776-783. 

29.  E.  Masry  and  D.  Tjostheim,  "Additive  Nonlinear  ARX  Time  Series  and  Projection  Estimates," 
Econometric  Theory,  Vol.  13  (April  1997),  pp.  214-252. 

30.  E.  Masry  and  J.  Fan,  "Local  Polynomial  Estimation  of  Regression  Functions  for  Mixing  Processes," 
Scandinavian  J.  of  Statistics,  Vol.  24  (June  1997),  pp.  165-179. 

31.  E.  Masry,  "Multivariate  Probability  Density  Estimation  by  Wavelet  Methods:  Strong  Consistency 
and  Rates  for  Stationary  Time  Series,"  Stochastic  Processes  and  Their  Applies.,  Vol.  67  (May  1997), 
pp.  177-193. 

32.  E.  Masry,  "Spectral  and  Multivariate  Probability  Density  Estimation  of  Continuous-Time  Stationary 
Processes  From  Randomly  Sampled  Data,"  invited  paper,  in  Applications  of  Time  Series  Analysis  in 
Astronomy  and  Meteorology,  T.  Subba  Rao,  M.B.  Priestley,  and  O.  Lessi,  Eds.,  London:  Chapman 
&  Hall,  1997.  pp.  89-102. 
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n.  DESCRIPTIVE  SUMMARY  OF  PROBLEMS  AND  RESULTS 

A  descriptive  summary  of  the  research  problems  studied  under  this  grant  and  the  nature  of  the 
results  obtained  is  now  presented  for  each  of  the  principal  areas  listed  in  Section  1. 

A.  Inference  for  Continuous-Time  Processes  from  Sampled  Data 

Research  in  this  area  focused  on  three  problems:  a)  spectral  estimation,  b)  prediction  and 
interpolation,  and  c)  Monte  Carlo  integration.  Our  contributions  consist  of  the  papers  [1][2][3][4][5][6]. 

a)  Spectral  Estimation. 

Let  X  =  {Y(r), -oo  <  r  <  00}  be  a  continuous-time  stationary  process,  {x^t}  be  the  sampling 
instants,  and  {X(Tjfe)}  be  the  discrete-time  observation  process.  We  are  interested  in  estimating  the 
statistical  structure  of  the  continuous-time  process  {X(r),  -  00  <  r  <  00}  from  a  finite  set  of  discrete-time 
observations  {X(Tjt),T;fe}2=i.  Particular  functions  of  interest  are  the  family  of  finite-dimensional 
distributions  and  densities  of  the  process  X,  the  correlation  function  C(r),  and  the  spectral  density  of 
the  process  X.  Clearly  if  the  the  sampling  instants  {x^}  are  equally-spaced,  consistent  estimates  of  the 
joint  densities  of  the  process  X  from  the  observations  {X(xjt),xjt}  is  not  feasible;  similarly,  if  the  process 
X  is  not  bandlimited,  consistent  estimates  of  C(r)  and  ^(A)  from  equally-spaced  observations  is  not 
possible  (due  to  aliasing)  unless  the  sampling  rate  is  allowed  to  diverge  to  infinity.  Our  main  goal  was  to 
identify  appropriate  nonequally-spaced  sampling  schemes  with  a  finite  average  sampling  rate  which 
would  allow  the  consistent  estimation  of  these  functions  as  the  number  of  observations  n  tends  to  infinity. 
Such  sampling  schemes  are  called  alias-free. 

We  considered  both  parametric  [1]  and  nonparametric  [2]  approaches:  Let  X  =  {X(r),-e>o<r  <00} 
be  a  continuous-time  stationary  process  with  spectral  density  where  ^  is  a  vector  of  unknown 

parameters.  For  example,  £  could  be  the  parameters  of  an  autoregressive  moving  average  continuous-time 
process.  Let  {x;fc}  be  a  stationary  point  process  on  the  real  line  which  is  independent  of  X.  We  considered 
in  [1]  the  identifiability  and  estimation  of  £  from  the  discrete-time  observations  {X(x;t),x*}2_i.  We 
established  the  consistency  and  the  joint  asymptotic  normality  of  appropriate  estimates  In  [2]  we 
provided  a  complete  statistical  analysis  of  suitable  estimates  of  the  (nonparametric)  spectral  density 
function  4)  (A).  We  identified  broad  classes  of  nonequally-spaced  sampling  schemes  {x^}  with  finite  mean 
sampling  rates  and  formulated  suitable  spectral  estimates  ^(A)  on  the  basis  of  the  discrete-time 
observations  {X(xt),Xfc}2=:i.  We  established  the  quadratic-mean  consistency  and  asymptotic  normality  of 
such  estimates  as  the  number  of  observations  n  tends  to  infinity.  It  was  shown  that  the  spectral  estimates 
are  consistent  for  ^  positive  values  of  the  average  sampling  rate.  No  such  results  could  exist  for  equally- 
spaced  samples.  In  [3]  we  provided  practical  guidelines  for  the  selection  of  appropriate  alias-free 
sampling  schemes  {xj^}  for  the  estimation  of  broadband  and  narrowband  spectra.  In  [4]  a  review  of 
available  results  for  the  estimation  of  the  covariance  function  C(t),  spectral  density  <I>(X),  and  the 
multivariate  probability  densities  of  continuous-time  processes  from  discrete¬ 

time  observations  is  presented. 
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b)  Interpolation  and  Prediction. 

One  of  the  classical  problems  in  signal  estimation  is  the  recovery  of  a  continuous-time  signal  f{t) 
from  a  sequence  of  its  samples  {f{tj)}.  The  well-known  Shannon’s  sampling  theorem  interpolates  a 
bandlimited  signal  from  equally-spaced  samples  taken  at  the  Nyquist  rate.  In  [5]  we  considered  the 
polynomial  interpolation  and  prediction  of  second  order  continuous-time  random  processes  from  a  finite 
number  of  randomly  sampled  observations  {X(r;)};  the  sampling  instants  {tj}  constitute  a  general 
stationary  point  process  on  the  real  line  which  is  independent  of  X.  We  establish  in  [5]  upper  bounds  on 
the  mean-square  interpolation  and  prediction  errors,  valid  for  ^  mean  sampling  rates  >0,  and  explored 
their  dependence  on  /3 ,  on  the  number  of  samples,  and  on  the  irregularity  of  the  sampling  instants. 

c)  Monte  Carlo  Integration. 

In  [6]  weighted  integrals  of  random  processes  are  approximated  by  the  trapezoidal  rule  based  on  a 
stratified  and  symmetrized  random  sample  of  size  n.  We  established  the  rate  of  convergence  to  zero  of  the 
mean-square  integral  approximation  error  as  the  sample  size  increases  indefinitely.  For  random  processes 
which  are  twice  mean-square  continuously  differentiable  it  is  shown  that  the  rate  is  n~^,  just  as  without  a 
random  component.  For  random  processes  which  are  a  bit  more  than  once,  but  not  twice,  mean-square 
continuously  differentiable  the  rate  is  shown  to  be  n~^.  In  both  cases  the  asymptotic  constant  is  also 
determined.  This  work  extends  the  results  of  Haber  [7][8]  for  deterministic  functions  to  Monte  Carlo 
integration  of  random  processes. 

B.  Wavelets,  Signal  Approximation,  and  Function  Estimation 

Wavelet  analysis  of  signals  is  a  rapidly  developing  area  of  research.  It  has  found  applications  in  a 
wide  variety  of  fields:  in  digital  signal  processing  because  of  its  connection  to  multirate  filtering;  in  image 
processing  because  of  applications  in  image  representation  and  compression  and  multiscale  edge 
detection;  in  numerical  analysis  because  of  applications  to  partial  differential  equations;  and  in  signal 
processing  (and  mathematical  physics)  because  of  the  time-frequency  (phase-space)  localization 
properties  of  the  wavelet  transform.  See  the  books  by  Meyer  [9],  Daubechies  [10],  and  Chui  [11]  and  the 
special  issue  of  the  IEEE  Transactions  on  Information  Theory  (March  1992). 

Our  contributions  in  this  area  consist  of  the  papers  [12][13][14][15][16].  We  focused  on  two  aspects 
of  wavelet  analysis:  a)  Approximation  of  functions  and  random  processes  via  orthonorraal  wavelet  bases, 
and  b)  the  use  of  wavelet-based  methods  for  the  estimation  of  multivariate  probability  density  functions. 


a)  Approximation  Results. 

In  [12]  we  developed  an  n!^  order  expansion  for  the  wavelet  approximation  error  at  resolution 
2  ^  for  functions  /  e  L^i-oo  ,  oo)  having  a  given  degree  of  smoothness  (say  n  derivatives).  Specifically, 
the  approximation  of  /  at  resolution  2“”*  is  given  by 

OO 

/m(0  “  2 

k--oo 

—  oo 

where  <l>m.kM  =  2~"^<l)(2'”x-k)  and  (Ji(a:)  is  the  scale  function.  The  exact  rate  of  convergence  and 


.*:=  J  fM<l>m.k(x)dx 
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asymptotic  constants  were  determined  and  their  dependence  on  /  and  on  the  scale  function  (j>  are 
obtained:  If  /  has  n  continuous  and  integrable  derivatives,  then  we  established  that 


£1.  +  ...+  ^2[/i/2] 

22/n  ^  ^ 


+  o{ 


2nm 


) 


where  Q  =  0  and  for  *  >  1, 


/  _  j\A:+l  00 

^2k  =  I  M2t  =  JJ(m-v)2*^(mX>(v)Jmc/v. 

-oo  -00 

Note  that  the  constants  M2*  depend  only  on  the  scale  function  (p .  When  n>2  and  M2  0,  then 

2  “  A^2  J  as  m  — >  00 

—oo 

and  this  rate  cannot  be  improved  by  additional  differentiability  of  /  beyond  second-order.  However,  faster 
rates  of  convergence  are  possible  if  certain  centered  moments  of  the  scale  function  (p  vanish.  These  results 
complement  and  refine  those  of  Mallat  [17,  Theorem  3]  and  Meyer  [9]  which  lead  to  an  upper  bound  on 
the  rate  of  convergence  of  the  error  . 

In  the  same  work  [12]  we  also  considered  the  wavelet  approximation  at  resolution  2””*  of  stationary 
and  nonstationary  second-order  random  processes.  All  stationary  and  most  nonstationary  second-order 
processes  do  not  have  sample  paths  in  L2(-oo ,  00)  and  thus  they  do  not  fit  the  standard  framework  of 
L2(-oo,oo)  wavelet  representation.  However,  with  probability  one,  the  sample  functions  of  mean-square 
continuous  stationary  and  nonstationary  random  processes  are  square  integrable  over  every  finite  interval. 
We  therefore  considered  in  [12]  the  wavelet  approximation  of  such  processes,  at  resolution  2“'",  over  a 
finite  interval,  say  [0,  T], 


—  2  ^mjt.  ^  m,k0) 
k--oo 

where  the  series  converges  in  L2  [0,  T]  with  probability  one  and 

00 

—  00 

and  we  provided  an  n'*  order  expansion  for  the  mean  integrated  squared  approximation  error, 

T 

elj  =  Ej 

0 

as  m  -4  00  under  very  mild  smoothness  assumptions  on  the  correlation  function  /?(r,j)  =  jE[X(t)X(5)]. 
The  precise  rate  of  convergence  as  well  as  the  asymptotic  constant  were  determined  in  [12].  These  results 
complement  and  refine  some  results  in  Cohen  et  al.  [18]  where  an  upper  bound  on  the  rate  of  convergence 
of  the  error  e^  j  is  given  under  a  condition  on  the  decay  at  infinity  of  the  spectral  density. 
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In  [13]  we  considered  the  approximation  of  Fractional  Brownian  motion  (FBM)  by  a  wavelet-based 
representation  at  resolution  2"'”.  Note  that  FBM  has  no  quadratic-mean  derivatives  and  hence  the  results 
we  obtained  in  [12]  are  not  applicable.  Fractional  Brownian  motion  is  a  widely  used  model  in  signal 
processing.  We  obtained  asymptotic  expressions  for  the  mean  integrated  squared  approximation  error  over 
finite  intervals. 

b)  Density  Estimation. 

Probability  density  estimation  is  a  fundamental  problem  in  statistical  data  analysis.  Recently, 
wavelet  methods  were  introduced  for  the  estimation  of  probability  density  functions.  The  first  papers  by 
Doukhan  and  L6on  [19],  Kerkyacharian  and  Picard  [20],  and  Walter  [21]  dealt  with  linear  wavelet 
estimators  in  i.i.d.  setting  and  established  upper  bounds  on  convergence  in  the  mean  Lp  norm.  Leblanc 
[22]  extended  the  work  of  Kerkyacharian  and  Picard  [20]  to  dependent  observations  again  establishing 
upper  bounds  in  the  mean  Lp  norm  estimation  error.  Donoho  et  al.  [23]  were  the  first  to  introduce 
nonlinear  wavelet  estimates  using  thresholded  empirical  wavelet  coefficients  and  established  minimax 
results  over  a  range  of  densities  belonging  to  the  Besov  space  ,5>0,l^p<oo,l<5^oo  and  a 
range  of  global  mean  Lp’  error  measures  with  \<p'<oo.  For  certain  values  of  p  and  p\  they  showed  that 
linear  estimates  have  a  suboptimal  rates  of  convergence. 

Our  contributions  in  this  area  consist  of  the  papers  [14][15].  In  [14]  we  established  precise  mean 
L2-norm  results  (rate  of  convergence  and  the  value  of  the  asymptotic  constant)  for  densities  in  the 
Sobolev  space  H2,s>0,  for  dependent  observations  using  linear  wavelet  density  estimates.  This 
improved  upon  the  earlier  works  where  only  an  upper  bound  on  the  error  was  obtained.  In  [15]  we 
considered  the  estimation  of  the  multivariate  probability  density  functions  f(x)  =  /(xj,  •  •  • ,  xj),d  >  1,  of 
the  random  process  {X,  }  using  linear  wavelet  methods.  We  established  the  strong  consistency  of  such 
estimates  along  with  rates  of  convergence  which  are  uniform  over  compact  subsets  of  R‘^.  The 
multivariate  density  function  f  is  assumed  to  belong  to  the  Besov  space  B^pg.  It  is  shown  that  these 
uniform  almost  sure  rates  coincide  with  the  best  attainable  rates  and  thus  nonlinear  thresholded  estimates 
(highly  hyped  in  the  literature)  provide  no  improvement  in  this  case.  It  should  be  noted  that  the  primary 
reason  for  the  claimed  performance  advantage  of  nonlinear  density  estimates  in  Donoho  et  al.  [23]  is  that 
the  adopted  performance  measure  there  is  mean  integrated  error.  This  advantage  disappears  when  a  sup- 
norm  error  is  used  as  the  performance  measure.  It  should  also  be  noted  that  for  rich  classes  of  functions 
such  as  the  Besov  class  B^  p  g  (where  functions  can  have  irregular  behavior),  a  measure  of  performance 
based  on  an  integrated  error  is  unsuitable:  Large  estimation  errors  can  occur  over  small  sets  in  which 
contribute  little  to  the  integrated  error;  a  sup-norm  performance  measure  is  clearly  much  more 
meaningful.  Also,  in  practice,  estimates  are  based  on  a  single  realization  and  thus  an  almost  sure 
convergence  rates  are  desirable.  Consequently  an  almost  sure  sup-norm  performance  measure  is  the 
appropriate  choice  for  estimating  density  functions  in  the  Besov  space  Bs,p^q  and  the  results  we  obtained 
in  [15]  imply  that  nonlinear  thresholded  density  estimates  have  no  real  advantage  over  linear  wavelet- 
based  density  estimates. 

In  [16]  we  considered  the  wavelet  transform  and  wavelet  coefficients  of  random  processes  with 
wide-sense  stationary  increments.  Their  covariance  functions  and  spectral  distributions  are  determined. 
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These  results  extend  the  work  of  Flandrin  [24]  for  the  special  case  of  fractional  Brownian  motion. 


C.  Neural  Networks  and  Function  Estimation 


The  research  in  this  area  focused  on  probability  density  and  regression  functions  estimation  using 
single  hidden  layer  sigmoidal  neural  networks.  Our  contribution  consists  of  two  papers  [25]  [26]. 

In  [25]  we  considered  the  estimation  of  a  probability  density  functions  f(x),  xeR^^.  we  constructed 
minimum  complexity  density  estimators  f„{x)  based  on  n  i.i.d.  observations  For  densities  / 

belonging  to  the  exponential  family,  we  establish  a  rate  of  convergence  for  the  expected  Hellinger 
distance, 


VS^-V7(i)  dx  =  0^^^ 


1 


1/2^ 


V  '  y 

The  important  point  here  is  that  the  rate  of  convergence  is  independent  of  the  dimension  d  in  sharp 
contrast  to  the  standard  methods  (kernel,  histogram,  nearest  neighbor  density  estimates)  which  suffer  from 
the  "curse  of  dimensionality"  whereby  the  rate  of  convergence  becomes  progressively  slower  as  the 
dimension  d  increases. 


In  [26]  we  considered  multivariate  regression  estimation  for  dependent  data  using  neural  networks. 
Let  }  be  a  bivariate  stationary  process  and  define  the  regression  function 

/•(x)4£[FiIXi  =  x], 

Minimum  complexity  estimation  of  r(x)  was  considered  previously  by  Barron  [27]  in  the  context  of  i.i.d. 
data.  In  [26]  we  extended  Barron’s  result  to  the  case  of  dependent  data:  For  strongly  mixing  processes 
with  geometric  decay,  we  showed  that  the  mean  integrated  squared  error  of  estimates  f(x),  based  on  the 
observations  •}"_!,  satisfies 


E  j[rn(x)-rix)fdPxix)  =  0 


l/2> 


Again  note  that  the  rate  of  convergence  is  independent  of  the  dimension  d  in  contrast  to  the  curse  of 
dimensionality  exhibited  by  standard  kernel  regression  estimates. 


D.  Probability  Density  and  Regression  Estimation  from  Noisy  Dependent  Data 

Probability  density  and  regression  estimation  play  a  central  role  in  communication  theory,  pattern 
recognition  and  classification  [28]-[29].  We  considered  the  problem  of  estimating  multivariate  probability 
densities  and  regression  functions  in  the  presence  of  additive  noise.  Our  contributions  in  this  area  are 
represented  by  papers  [30][31][32][33][34][35]. 

a)  Density  Estimation 

Let  be  a  stationary  process  and  for  each  integer  p  >  1  let  /°(x ;  p)  =  /°(xi,  •  •  • ,  p) 

be  the  joint  probability  density  function  of  the  random  variables  X°,  •  •  • ,  Xp.  Consider  the  deconvolution 
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problem 

Xi  =  Xf+ei 

where  the  (noise)  process  {£|  }n:-oo  consists  of  i.i.d.  random  variables  with  known  density  h(x), 
independent  of  the  process  {X°}^_oo-  Given  the  noisy  observations  one  desires  to  estimates  the 

multivariate  density  f°(x;  p).  This  is  clearly  a  probability  density  deconvolution  problem.  It  arises  in 
biological  studies  (Medgyessy  [36],  Mendelsohn  and  Rice  [37]),  communication  theory  (Wise  et  al  [38], 
Snyder  et  al  [39]),  and  applied  physics  and  analytical  chemistry  (Jones  and  Misell  [40],  Harder  and  Galan 
[41]).  The  special  case  of  i.i.d.  observations  (X,°  are  i.i.d.  random  variables  and  p  =  l)  received 
considerable  attention  in  the  literature  with  contributions  by  Carroll  and  Hall  [42],  Liu  and  Taylor  [43], 
Stefanski  and  Carroll  [44],  Zhang  [45],  and  Fan  [46]. 

In  a  series  of  papers  [30]  [32]  [33]  we  extended  the  above  works  in  the  following  directions: 

•  The  process  is  a  stationary  mixing  process. 

•  Estimation  of  ^the  joint  probability  density  functions  of  the  process  {Xf}11_^  is  considered. 

In  [30]  we  established  the  quadratic-mean  convergence  of  deconvolution  estimators  f^(x;  p)  along 
with  rates  of  convergence  for  processes  satisfying  a  variety  of  mixing  conditions.  In  [33]  we  established 
sharp  rates  of  almost  sure  convergence  for  the  estimators  }n^x\  p).  In  [32]  we  derived  the  asymptotic 
normality  of  the  estimators  fni.x\  p).  In  all  three  papers  the  dependence  of  the  rate  of  convergence  on  the 
smoothness  of  the  noise  density  h{x)  is  investigated.  Generally,  the  smoother  the  noise  density,  the  slower 
is  the  rate  of  convergence.  Thus,  for  example,  when  the  noise  is  Gaussian,  which  is  super  smooth,  the 
rates  of  convergence  are  quite  slow. 

b)  Regression  Estimation 

Let  and  be  jointly  stationary  processes  and  {e,}  a  sequence  of  i.i.d.  random 

variables,  independent  of  the  processes  [X?}n=-eo  with  a  common  probability  density 

h{x).  Put 

X,-  =  Xf  +  e,-  ,  t  =  0, ±  1,  ••• . 

Let  G{x)  be  an  arbitrary  function  on  the  real  line,  not  necessarily  bounded  or  continuous,  and  assume  that 
E\G{Y{)  I  <  oo.  Define  the  multivariate  regression  function  by 

m{x-,p)  =  E[G(yp)\X§  =  x] 

where 


Xo°  =  (Xf  ,---,x;). 

Given  the  noisy  observations  {Xy,  Y one  desires  to  estimate  m{x',  p).  The  problem  of  regression 
with  errors-in- variables  arises,  for  example,  in  medical  and  epidemiologic  studies  where  risk  factors  are 
partially  available,  see  Prentice  [47]  and  Whittemore  and  Keller  [48]. 

In  [34]  [35]  we  provided  a  thorough  analysis  of  the  multivariate  regression  problem  with  errors  in 
variables  for  dependent  data.  In  [35]  we  obtained  sharp  rates  of  almost  sure  convergence  for  the 
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deconvolution  estimates  of  mix;  p)  for  strongly  mixing  processes.  In  [34]  we  established  the  asymptotic 
normality  of  these  estimates  for  a  variety  of  mixing  processes.  These  results  significantly  extend  the 
earlier  works  by  by  Fan  and  Truong  [49]  who  examined  the  special  case  of  i.i.d.  observations  (the 
variables  {X,°,  7,}  are  i.i.d.,  p=l  and  G(x)  =  x). 

E.  Nonlinear  Time  Series 

Nonlinear  time  series  is  a  very  active  area  of  research  in  general  [50]  [51]  and  in  econometrics  in 
particular  [52]  [53].  Among  the  models  considered  in  the  literature  are  the  autoregressive  conditional 
heteroscedastic  (ARCH)  model  [54]  which  had  a  substantial  impact  on  modeling  of  econometric  time 
series  and  the  autoregressive  with  exogenous  variables  (ARX)  model.  Our  contributions  in  this  area 
consist  of  the  papers  [55]  [56]. 

The  ARCH  model  has  the  representation 

~  Sli^t—q’  ■ ' ' » ' '  ■ » l) 

where  g2^0  and  the  innovations  [e,}  are  i.i.d.  random  variables  with  zero  mean  and  variance  A 
primary  research  focus  has  been  the  identification/estimation  of  the  nonlinear  functions  gi  and  g2  from 
data.  When  gj  and  g2  are  specified  in  a  parametric  form  (e.g.  gj  being  linear  and  g2  being  quadratic), 
estimates  of  their  parameters  were  considered  in  [54].  In  [55]  we  provided  a  thorough  analysis  of  the 
convergence  properties  of  nonparametric  estimates  of  gj  and  of  cy^g2  including  sharp  rates  of  almost  sure 
convergence  and  asymptotic  normality. 

In  [56]  we  considered  the  identification/estimation  of  the  functional  structure  of  the  bivariate 
nonlinear  ARX  time  series 

=  83iYt-qy  •  •  • .  +  g4i^t-p>  ’  ’ ' .  X,)  +  e, 

Xf  —  gsi^t-pt  *  ■  ‘ » Xj_i)  +  £  j  . 

where  (e,)  and  {e,}  are  independent  series  each  consisting  of  zero  mean  i.i.d.  variables  with  variances 
and  <Tg  respectively.  There  is  an  extensive  literature  on  parametric  ARX  models  whereby  the  functional 
structure  of  the  g[  is  known  except  for  finite  number  of  parameters  and  a  rigorous  theory  is  available  (see, 
for  example,  Pdtscher  and  Pracha  [57]  [58]).  No  rigorous  theory  was  available  in  a  nonparametric  setting. 
Using  the  projection  idea  of  Auestad  and  Tjostheim  [59]  we  provided  in  [56]  a  comprehensive  analysis  of 
suitable  estimates  of  the  functions  g^  and  g4  including  rates  of  weak  convergence  and  asymptotic 
normality.  An  important  consequence  of  these  result  is  the  dimension  reduction  in  the  sense  that  the  rate 
of  convergence  for  the  estimate  of  g^  involves  only  its  dimension  q  rather  than  p  +  q  obtained  normally 
by  standard  regression  estimates.  Similarly  for  g4. 

E  Local  Polynomial  Fitting 

Local  polynomial  fitting  is  a  new  methodology  for  estimation  of  regression  functions  which 
dramatically  outperforms  the  classical  kernel  approach  of  Nadaraya  [60]  -  Watson  [61].  The  local 
polynomial  fitting  approach  was  introduced  originally  by  Stone  [62]  and  studied  by  Cleveland  [63], 
Tsybakov  [64],  Fan  [65]  [66]  and  many  others.  Local  polynomial  fitting  has  significant  advantages  over 
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the  Nadaraya-Watson  regression  estimator:  For  local  linear  fitting  it  has  been  shown  to  reduce  the  bias 
(see  Chu  and  Marron  [67]  and  Fan  [65]);  it  adapts  automatically  to  the  boundary  of  design  points  (see 
Fan  and  Gijbels  [68],  Ruppert  and  Wand  [69]  —  No  boundary  modification  is  required.  It  is  superior  to 
the  Nadaraya-Watson  estimator  in  the  context  of  estimating  the  derivatives  of  the  regression  function  (see 
Fan  and  Gibels  [68]  and  Ruppert  and  Wand  [69]).  All  these  works  consider  i.i.d.  setting  and  either  linear 
or  quadratic  polynomial  fitting. 

During  the  current  grant  period  we  embarked  on  a  major  project  to  extend  these  works  to  more 
general  and  useful  situations:  Let  be  jointly  stationary  processes  and  let  the  regression  function 

m{x\ ,  •  •  • ,  Xp)  be  defined  by 

mixi,...,Xp)^E{Yp\Xi=Xx,-',Xp  =  Xp\. 

For  example,  when  F,-  =  X,+r,  the  regression  function  m  gives  the  best  r-step  predictor  of  the  process 
{X,}.  In  a  series  of  papers  [  70][71][72]  we  formulated  and  established  the  convergence  properties  of 
local  polynomial  fitting  of  arbitrary  order  ^  >  1  (not  just  q-\  and  q  =  2).  Moreover,  we  allowed  the 
processes  (F,-,  X,  }  to  be  general  dependent  processes  rather  than  i.i.d.  pairs.  In  [70]  we  established  sharp 
rates  of  almost  sure  convergence  for  the  local  polynomial  estimates  of  the  regression  function 
m(xi,...,Xp)  and  all  its  partial  derivatives  up  to  a  total  order  q.  In  [71]  we  established  the  joint 
asymptotic  normality  of  these  estimates  along  with  asymptotic  expressions  for  their  bias  and  variance. 

G.  Spread  Spectrum  Communication  Systems 

Spread  Spectrum  communication  systems  offer  immunity  against  narrow-band  interference  [73]. 
There  are  various  techniques  for  enhancing  this  immunity  including  interference  suppression  filters  and 
frequency  hopping  methods. 

In  [74]  [75]  we  provided  a  comprehensive  analysis  (and  design)  of  fast  frequency-hopped  spread 
spectrum  communication  system  employing  differential  binary  phase  shifted  keying  (DBPSK) 
modulation  and  differentially  coherent  modulation.  The  receiver  utilizes  a  hop  timing  tracking  loop  to 
"lock"  the  hop  clock.  In  [74]  the  performance  characteristics  of  the  tracking  loop  is  studied.  In  [75]  we 
obtained  the  bit  error  probability  of  the  overall  system  when  it  is  operating  under  additive  white  Gaussian 
noise  plus  partial-band  jamming.  Our  modeling  and  results  extend  those  in  the  literature  [76]  where  the 
hop  timing  tracking  errors  are  ignored. 

In  [77]  we  considered  the  problem  of  intercepting  a  direct  sequence  spread  spectrum  signal  which  is 
embedded  in  both  narrowband  interference  and  additive  white  Gaussian  noise.  We  employed  a 
compression  receiver  the  heart  of  which  is  a  surface  acoustic  wave  (SAW)  device  for  interference 
rejection.  A  central  limit  theorem  is  established  for  the  receiver’s  output  which  forms  the  basis  for 
analyzing  the  performance  of  the  receiver.  Numerical  results  are  presented  to  illustrate  the  receiver’s 
operating  characteristics 


H.  Other  Contributions 

Additional  contributions  [78]  [79]  [80]  [81]  [82]  were  made  in  a  broad  range  of  topics.  We  briefly 
sununarize  the  contributions  of  the  more  important  ones  [78]  [80]  [82]. 
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In  [78]  we  considered  recursive  probability  density  estimation  for  vector-valued  processes  {Xj} 
from  dependent  samples  We  established  rates  of  almost  sure  convergence  for  the  global  Lp  error 

R‘‘ 

for  p  =  1, 2.  We  obtained  results  for  both  ergodic  processes  and  mixing  processes.  In  addition,  the  notion 
and  properties  of  Hilbert  space- valued  mixingales  are  developed  and  strong  laws  of  large  numbers  are 
given.  The  results  on  Hilbert  space-valued  mixingales  extend  those  of  McLeish  [83]  who  dealt  with  real- 
valued  mixingales.  The  results  on  density  estimates  extend  the  point-wise  convergence  results  of  Masry 
[84]  and  Tran  [85]  which  were  obtained  under  considerably  stronger  conditions. 

In  the  area  of  parametric  spectral  estiniation  we  considered  in  [80]  the  problem  of  estimating  the 
spectral  density  (j)  (A.)  of  an  autoregressive  (AR(p))  stationary  process 

/=1 

from  a  finite  set  of  noisy  observations. 

Yj  =  Xj  +  Wj. 

This  is  a  classical  and  practically  important  problem  in  signal  processing.  A  modified  spectral  estimator 
based  on  the  high-order  Yule-Walker  equations  was  considered.  Sharp  rates  of  almost  sure  convergence 
are  established  for  the  estimates  of  the  autoregressive  parameters,  the  innovation  variance,  the  additive 
noise  variance,  and  the  spectral  density  (j)(A,)  of  the  AR  process  {X,  }.  No  almost  sure  rates  of  convergence 
were  previously  available.  The  work  supplements  the  asymptotic  normality  result  obtained  earlier  by 
Gingras  and  Masry  [86]  and  the  work  of  Pagano  [87]  where  nonlinear  regression  method  is  used.  The 
advantage  of  using  the  (linear)  high-order  Yule- Walker  equations  is  the  simplicity  of  implementation. 

Adaptive  linear  estimation  methods  based  on  the  principle  of  steepest  descent  and  its  variations  have 
been  applied  to  a  wide  range  of  problems  such  as  filtering,  noise  canceling,  line  enhancement,  antenna 
processing,  and  interference  suppression.  The  well  known  LMS  algorithm  has  been  extensively  studied  in 
the  literature.  One  of  the  most  popular  and  widely  used  algorithms  is  the  sign  algorithm  whose  recursive 
equations  are  as  follows:  The  updated  equation  for  the  vector  h(;)  of  the  estimated  filter’s  coefficients  at 
iteration  j  is  given  by 

h(;  +  l)  =  h(;)  +  fix(;)sgn[e(j)],  j  =  l,2,--- 

where  x(j)  is  the  data  at  iteration  j,  e(j)  is  the  error  in  estimating  the  desired  signal  d(j)  using  the  data 
vector  x(  j), 

and  fi  is  the  adaptation  size.  Here  h(J)  and  x(j)  are  column  vectors  with  dimension  N.  The  popularity  of 
the  sign  algorithm  is  due  to  its  simplicity  of  implementation  in  that  only  the  polarity  of  the  error  e(j)  is 
used  to  update  the  estimate.  One  is  concerned  with  the  convergence  of  the  sign  algorithm  as  7  — >  oo:  If 
hopt  is  the  optimal  Wiener-Hopf  filter  with  corresponding  minimum  mean-square  error  £^n,  then  one 
seeks  the  convergence  properties  of  the  deviation  error 
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yU)  =  Hj)-Kpi 

and  of  the  signal’s  estimation  error  e{j)  as  j  — >  oo.  For  a  fixed  step  size  /x  >  0,  rigorous  convergence 
analysis  was  first  given  in  Gersho  [88];  heuristic  results  were  later  given  in  [89][90][91].  Asymptotic 
results,  as  the  step  size  p  ->  0  were  established  in  [92]. 

In  [82]  we  established  asymptotic  time-averaged  convergence  for  the  mean-square  deviation  error 
E[  llv(7)ll^]  and  for  the  signal  estimation  error  E[e^(j)]  under  the  assumption  of  i.i.d.  Gaussian  data. 
Specifically  we  proved  that  for  any  initial  weight  vector  h(l)  and  for  any  positive  step  size  u  we  have 

limsup  -  X  llv(7)l^]  ^Cin  +  . 

/l-^OO  ^  j=l 

Moreover,  we  showed  that 

n  ;=1  n 

with 

J  n 

limsup  “  2  ^u)  ^  C3  JU  +  C4]li^ 

n—¥oo  w  y=l 

where  the  constants  C,-  are  specified  in  [82].  Thus  the  time-averaged  mean-square  deviation  error  and  the 
excess  signal  estimation  error  are  proportional  to  the  step  size  p  which  is  similar  to  the  behavior  of  the 
LMS  algorithm. 
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